Complex systems involving a large number of variables may be influenced by a variety of factors. Models have been developed to try to measure or quantify the impact of individual factors. However, these models can't control for the influence of additional factors. As a result, the validity or accuracy of these models, and the validity or accuracy of the valuations or predictions generated by these models, varies. These models are measurement tools that can be calibrated and tuned. In order to calibrate a measurement tool, the tool should be applied to a standard of known accuracy. However, for many complex systems, there is no known repeatable standard.
This problem is seen, for example, in models used to evaluate marketing strategies and identify how changes in a marketing strategies impact key performance indicators. Marketing models, such as media mix models (“MMM”), typically analyze aggregated historical data representing real world events that can't be recreated (e.g., because exterior variables are beyond the modeler's view or control). What is needed is a reliable market simulation that can establish a ground truth for evaluating marketing models that rely on aggregate historical data.